Multi-Modal Definite Clause Grammar

This paper describes the first reported grammatical framework for a nmltimodal interface. Although multimodal interfaces offer the promise of a flexible and user fl'iendly means of human-coml)uter interaction, no study has yet appeared on formal granunatical f'l'ame-works for theln. We have developed Multi-Modal Definite Clause Ch'ammar (MM-I)CG), an extension of Definite Clause Gramumr. The major features of MM-I)CG inch, de eal)ability to handle an arbitrary mlmber of modes and temporal information in grammar rules, l:ur-ther, we have developed MM-DCG translator to transfer rules in MM-DCG into Prolog predicates. 1 Introduction This paper describes tile first reported grammatical fi'amework for a multimodal interface. Specifically, the authors have developed MM-DC.G (Multi-Modal l)cfi-nite Clause Gra,nmar), an extension of I)CCI [Pereira and Warren, 1980] for lnultm3odal input processing. The major features of MM-DCG include capability to handle an arbitrary nn,nber of modes and temporal information in grammar rules. The motivation behind this research was two-foht. First, the extension to multimodal was found t.o be the minimum requirement [br natural language interface systems to be insta.lled in real al~plications. We have developed natural language interface for relational database (RDB) [Shimazu et. al., I9!)2; Arita et. al., 1992a; Arita et. al., 1992b]. Spoken user queries are transformed into SQL specifications, and dispatched t.o RDBMS. The retrieved results are displayed at a computer terminal. The results include not only table forms but also picture images, like Figure 1. When users see picture images on the terminal, they naturally want to generate following queries by referring to such picture images. For example, they want to say, "Show me the interior of this one" or "Are there the same type of cars as this ear" while pointing at a specific picture on the display. If such multi-modal utterances be accept.able, the natural language interface will be more practical Figure I: Natural Language Interface Screen hnage enough to be used in many real world applications. Second, no st;udy has yet appeard on developing for-real grammatical fi'amework for multi-modal interfaces. Although there have been many researches on multi-modal systems, these systems are built as task-specific expert systems. The capability of such systems to process multi-moda.l inputs is too limited to interpret complex multi-modal expressions. This is mainly due to the fact that they have not developed their systems on for-nml grammatical framework for multi-modal interfaces. MM-DCG is the first reported grammatical framework for a multimodal interface. Multi-modal input processing rules can …